Add ASP.NET Core perf-build pipeline#5243
Draft
LoopedBard3 wants to merge 3 commits into
Draft
Conversation
Stand up an Azure DevOps perf-build pipeline hosted in dotnet/performance that builds every commit on dotnet/aspnetcore main from source, packs per-RID Microsoft.AspNetCore.App.Runtime archives, and uploads them to the Build Cache Service for dotnet/crank bisection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new Azure DevOps perf-build pipeline in dotnet/performance to build dotnet/aspnetcore main commits from source, pack per-RID Microsoft.AspNetCore.App.Runtime archives, and upload/register them in BCS using the existing shared templates (extended to support a caller-supplied SHA).
Changes:
- Introduces
eng/pipelines/aspnetcore-perf-build.yml+eng/pipelines/aspnetcore-perf-build-jobs.ymlto build/pack/publish ASP.NET Core runtime-pack artifacts for the locked set of 5 config keys. - Adds
shaparameter plumbing through the shared register/upload dispatcher + leaf templates so non-self(resource) builds can write BCS blobs under the triggering repo commit. - Adds
eng/pipelines/tools/pack-bcs-archives.ps1to produce the BCS archive layout from runtime-pack nupkgs.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| eng/pipelines/aspnetcore-perf-build.yml | New root pipeline with aspnetcore repo-resource trigger and Register/Build/Upload stages. |
| eng/pipelines/aspnetcore-perf-build-jobs.yml | New authored build jobs to build aspnetcore from source (Windows multi-arch + Linux x64/arm64) and publish artifacts. |
| eng/pipelines/tools/pack-bcs-archives.ps1 | New packaging script to turn runtime-pack nupkgs into BCS archive layout and validate contents. |
| eng/pipelines/upload-build-artifacts-jobs.yml | Adds sha parameter and threads it through aspnetcore upload branches. |
| eng/pipelines/templates/upload-build-artifacts-job.yml | Adds sha parameter and uses it in the BCS blob path for uploads. |
| eng/pipelines/register-build-jobs.yml | Adds sha parameter and forwards it into per-buildType register jobs. |
| eng/pipelines/templates/register-build-job.yml | Adds sha parameter and uses it in the BCS blob path for buildInfo.json. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+135
to
+139
| - stage: Build | ||
| displayName: Build | ||
| dependsOn: [] | ||
| jobs: | ||
| - template: /eng/pipelines/aspnetcore-perf-build-jobs.yml |
Comment on lines
+121
to
+125
| -Configuration Release | ||
| -arch arm64 | ||
| -pack | ||
| -noBuildJava | ||
| -noBuildNative |
Comment on lines
+6
to
+11
| .DESCRIPTION | ||
| This is the single, canonical "find the runtime-pack nupkg and zip it into | ||
| the BCS archive layout" recipe used by every job in perf-build.yml (Windows | ||
| x64/x86/arm64, Linux x64, Linux arm64). The build jobs invoke it from their | ||
| `afterBuild` hook with the RID(s) they produced and the archive format that | ||
| matches their platform convention (.zip on Windows, .tar.gz on Linux). |
Comment on lines
+42
to
+46
| .EXAMPLE | ||
| ./pack-bcs-archives.ps1 -Rids win-x64,win-x86,win-arm64 -Format zip | ||
|
|
||
| .EXAMPLE | ||
| ./pack-bcs-archives.ps1 -Rids linux-x64 -Format targz |
Add an early, unconditional RegisterBuild job that tags the run with aspnetcore-sha:<40-char sha> (the aspnetcore commit from the repo-resource version macro). The build record's sourceVersion is the performance commit, so this tag is the dotnet-performance-infra indexer's sole aspnetcore-sha signal. Additive; the BCS {sha} override and buildInfo.json are unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…version)
The dotnet-performance-infra indexer derives the aspnetcore commit directly from the build's resources.repositories.aspnetcore.version via the runs API, so the dedicated TagBuild job is redundant. The BCS {sha} path override and buildInfo.json are unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LoopedBard3
changed the title
Comment on lines
+60
to
+65
| variables: | ||
| - _AspNetCoreRoot: $(Agent.BuildDirectory)\s\aspnetcore | ||
| - _PackScript: $(Agent.BuildDirectory)\s\performance\eng\pipelines\tools\pack-bcs-archives.ps1 | ||
| - _ShippingDir: $(Agent.BuildDirectory)\s\aspnetcore\artifacts\packages\Release\Shipping | ||
| - _StagingRoot: $(Build.ArtifactStagingDirectory)\bcs | ||
| - MSBUILDUSESERVER: "1" |
Comment on lines
+182
to
+186
| variables: | ||
| - _AspNetCoreRoot: $(Agent.BuildDirectory)/s/aspnetcore | ||
| - _PackScript: $(Agent.BuildDirectory)/s/performance/eng/pipelines/tools/pack-bcs-archives.ps1 | ||
| - _ShippingDir: $(Agent.BuildDirectory)/s/aspnetcore/artifacts/packages/Release/Shipping | ||
| - _StagingRoot: $(Build.ArtifactStagingDirectory)/bcs |
Comment on lines
+240
to
+244
| variables: | ||
| - _AspNetCoreRoot: $(Agent.BuildDirectory)/s/aspnetcore | ||
| - _PackScript: $(Agent.BuildDirectory)/s/performance/eng/pipelines/tools/pack-bcs-archives.ps1 | ||
| - _ShippingDir: $(Agent.BuildDirectory)/s/aspnetcore/artifacts/packages/Release/Shipping | ||
| - _StagingRoot: $(Build.ArtifactStagingDirectory)/bcs |
Comment on lines
+121
to
+126
| -Configuration Release | ||
| -arch arm64 | ||
| -pack | ||
| -noBuildJava | ||
| -noBuildNative | ||
| /p:OnlyPackPlatformSpecificPackages=true |
Comment on lines
+10
to
+14
| # sha controls the {sha} segment of the BCS blob path. Defaults to the self | ||
| # repo commit ($(Build.SourceVersion)) for back-compat with the runtime | ||
| # perf-build pipeline. Resource-built tenants (e.g. aspnetcore, whose source | ||
| # is a repository resource rather than self) pass the resource's commit, e.g. | ||
| # sha: $(resources.repositories.aspnetcore.version). Runtime config branches |
Comment on lines
+114
to
+118
| # The aspnetcore commit this build corresponds to is NOT tagged on the run: the | ||
| # dotnet-performance-infra indexer reads it directly from the build's | ||
| # `resources.repositories.aspnetcore.version` via the runs API (intrinsic to the | ||
| # build carrying the aspnetcore repo resource). The BCS {sha} path override below | ||
| # still uses that same value. |
Comment on lines
+38
to
+39
| # ATOMICITY: continueOnError:false everywhere; any failure sinks the build to | ||
| # Failed and the MissingBuildsTrigger Function skips indexing that SHA. |
Open
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Stands up a new Azure DevOps perf-build pipeline hosted in dotnet/performance that builds every commit on dotnet/aspnetcore
mainfrom source, packs per-RIDMicrosoft.AspNetCore.App.Runtimearchives, and uploads them to the Build Cache Service (BCS @pvscmdupload) so dotnet/crank can resolve ASP.NET Core runtime binaries by commit SHA for performance-regression bisection.This moves work that currently lives in dotnet/aspnetcore (a
perf-build.ymlto be retired there) into dotnet/performance.Why move it here
Team merge control. Hosting the pipeline in dotnet/performance gives the perf team full ownership of the build/upload contract instead of it living in aspnetcore.
What's in the change
New
eng/pipelines/aspnetcore-perf-build.yml— root pipeline: 5 per-config boolean params, theaspnetcorerepo-resource trigger, and the 3 stages (RegisterBuild / Build / UploadArtifacts).eng/pipelines/aspnetcore-perf-build-jobs.yml— authored build jobs (Windows x64/x86/arm64 in one job; Linux x64; Linux arm64).eng/pipelines/tools/pack-bcs-archives.ps1— moved verbatim from aspnetcore; the canonical "find runtime-pack nupkg → produce the lowercasemicrosoft.aspnetcore.app.runtime.{rid}/Release/...archive" recipe.Edited (shared runtime templates — back-compat preserved)
register-build-jobs.yml,templates/register-build-job.yml,upload-build-artifacts-jobs.yml,templates/upload-build-artifacts-job.yml— added ashaparameter (default$(Build.SourceVersion)), threaded through to the BCS blob path. The 5aspnetcore_*upload branches (added in Generalize BCS upload + register-build templates with repoName parameter #5241) now threadshatoo.Key design decisions / things to scrutinize
Can't reuse aspnetcore's
default-build.ymlcross-repo. That template has hard@selfreferences, and@selfalways resolves to the root pipeline repo (= performance), which doesn't carry aspnetcore'seng/commoncontract. So the build jobs are rebuilt from source here as plain- job:blocks that invoke aspnetcore's owneng/build.cmd/eng/build.sh.SHA inversion (load-bearing). Because the pipeline is hosted in performance,
Build.SourceVersionis the performance commit, not the aspnetcore one. The correct BCS{sha}is the triggering aspnetcore commit =$(resources.repositories.aspnetcore.version). The root pipeline passes that assha:to the register/upload templates. The shared templates defaultshato$(Build.SourceVersion)so the existing dotnet/runtime perf-build is unaffected.Build↔commit correlation (Option B). The build record's sourceVersion is performance's commit, so the dotnet-performance-infra indexer does not rely on sourceVersion. Instead it derives the aspnetcore commit directly from the build's
resources.repositories.aspnetcore.versionvia the runs API — intrinsic to every build that carries the aspnetcore repo resource, so no pipeline-side run tagging is needed. This avoids the silent-empty-latestBuildsfailure mode a forgotten or hand-edited build tag could otherwise introduce.Repo-resource CI trigger for per-commit firing. The aspnetcore AzDO mirror
internal/dotnet-aspnetcoreis atype: gitrepository resource withtriggeronmain(batch: false→ eager, per-commit). A push to aspnetcore main fires this pipeline (Build.Reason == ResourceTrigger); jobs check out that resource at the triggering commit. (Note: stage gating usesResourceTrigger, notIndividualCI— another consequence of the host inversion vs. the aspnetcore reference.)Build from PUBLIC feeds. No internal runtime download, no
enable-internal-runtimes/get-delegation-sasstep, nodotnetbuilds-internal-readconnection. Proof: aspnetcore's ownci-public.ymlbuilds every public PR with_InternalRuntimeDownloadArgsempty. Only the BCS upload uses the proven.NET Performanceconnection.Per-config booleans for forward-compat indexing. 5 boolean params (default true) gate their build jobs and feed the register/upload
buildTypearrays, so the dotnet-performance-infraMissingBuildsTriggerPerConfiguration indexer can count exactly these keys. In v1's eager mode all 5 are always built; the booleans let a future Function queue a subset.No 1ES. Per-commit perf artifacts go to a private BCS cache (unsigned, not redistributed), so we don't extend
1ES.Official— matching dotnet/runtime's perf-build.Atomicity.
continueOnError: falseeverywhere; any failure sinks the build to Failed and the indexer skips that SHA. UploadArtifactsdependsOn: [Build, RegisterBuild],condition: succeeded().The 5 configs / RIDs (locked)
aspnetcore_x64_linuxLinux_x64_buildaspnetcore_arm64_linuxLinux_arm64_buildaspnetcore_x64_windowsWindows_buildaspnetcore_arm64_windowsWindows_buildaspnetcore_x86_windowsWindows_buildBCS layout:
builds/aspnetcore/buildArtifacts/{sha}/{configKey}/{buildInfo.json,archive}.Validation status
The full pipeline can't be run locally. All 7 YAML files parse; job-name ↔
dependencyJobName, artifact names, and pack-script output paths were cross-checked for consistency. Real validation = a manual queued run after merge (pool image availability, aspnetcore from-source build on the dnceng internal pool, end-to-end BCS upload). The aspnetcore-hosted version will be retired once this is validated.